feat: add coverage report generation with cargo-llvm-cov by dblnz · Pull Request #1301 · hyperlight-dev/hyperlight

dblnz · 2026-03-11T14:57:43Z

Add coverage infrastructure for the Hyperlight project:

Justfile: add coverage, coverage-html, coverage-lcov, and coverage-ci recipes using cargo-llvm-cov for LLVM source-based code coverage
CI: add Coverage.yml weekly workflow (Monday 06:00 UTC, manual trigger) running on kvm/amd with self-built guest binaries
.gitignore: exclude coverage/ output directory
docs: add how-to-run-coverage.md with local and CI usage instructions

Guest/no_std crates are excluded from coverage because they define #[panic_handler] and cannot be compiled for the host target under coverage instrumentation. Coverage targets host-side crates only.

This PR:

only runs this on linux with KVM

Add coverage infrastructure for the Hyperlight project: - Justfile: add coverage, coverage-html, coverage-lcov, and coverage-ci recipes using cargo-llvm-cov for LLVM source-based code coverage - CI: add Coverage.yml weekly workflow (Monday 06:00 UTC, manual trigger) running on kvm/amd with self-built guest binaries - coverage-ci mirrors test-like-ci by running multiple test phases with different feature combinations (default, single-driver, crashdump, tracing) and merging profdata into a single unified report - Coverage summary is displayed in the GitHub Actions Job Summary for quick viewing; full HTML report is downloadable as an artifact - docs: add how-to-run-coverage.md with local and CI usage instructions Guest/no_std crates are excluded from coverage because they define coverage instrumentation. Coverage targets host-side crates only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

jsturtevant · 2026-03-11T23:42:39Z

docs/how-to-run-coverage.md

+
+1. Builds guest binaries (`just guests`)
+2. Installs `cargo-llvm-cov`
+3. Runs `just coverage-ci kvm` — this mirrors `test-like-ci` by running multiple test phases with different feature combinations and merging the results into a single coverage report


should we cover other hypervisors?

We could do this. I'll see if I include it in this PR or in a follow-up one, but I think it only needs some additional runners in the workflow to do that. I'll check it out

jsturtevant · 2026-03-11T23:47:39Z

Justfile

+    cargo llvm-cov {{ coverage-packages }} --no-default-features --features build-metadata,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report
+
+    @# Phase 3: crashdump feature tests
+    cargo llvm-cov {{ coverage-packages }} --no-default-features --features crashdump,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report -- test_crashdump


some of the examples have tests too, like --example crashdump should those be included? Do examples count as coverage?

I would say probably not

I am not sure.

As they exercise various areas of our code, I think we should run everything we have (tests + examples) and aggregate them in a single report so that we have a broad view of the degree of coverage we have.

jsturtevant

Its nice to see a report. Do you have thoughts on how do we ensure we are looking at this report and not just another artifact? Do we post an issue on run? post in release? Review at community meetings?

I ran it locally and I am seeing 70.42%, which quite high! Its nice to see this. I am not to worried about minor changes but would like to see it trending even or even improving over time.

ludfjig

This is awesome, I also tried it locally and works great. I only have one question: is this the recommended way to generate coverage reports (using cargo-llvm-cov), or are there alternative ways? I'm only asking because I am not familiar with this area.

ludfjig · 2026-03-12T02:39:49Z

Justfile

+    cargo llvm-cov {{ coverage-packages }} --no-default-features --features build-metadata,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report
+
+    @# Phase 3: crashdump feature tests
+    cargo llvm-cov {{ coverage-packages }} --no-default-features --features crashdump,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report -- test_crashdump


I would say probably not

ludfjig · 2026-03-12T03:06:34Z

Justfile

+# generate a text coverage summary to stdout (run `just guests` first to build guest binaries)
+coverage: ensure-cargo-llvm-cov
+    cargo llvm-cov {{ coverage-packages }}
+
+# generate an HTML coverage report to target/coverage/html/ (run `just guests` first to build guest binaries)
+coverage-html: ensure-cargo-llvm-cov
+    cargo llvm-cov {{ coverage-packages }} --html --output-dir target/coverage/html
+
+# generate LCOV coverage output to target/coverage/lcov.info (run `just guests` first to build guest binaries)
+coverage-lcov: ensure-cargo-llvm-cov
+    mkdir -p target/coverage
+    cargo llvm-cov {{ coverage-packages }} --lcov --output-path target/coverage/lcov.info


Does running these 3 in sequence re-run the tests 3 times? It would be nice if we could run once, and then you can choose what format to generate the report after (maybe separated into 2 different just-recipes)

Yeah, there's an option to select the type of report at the end of the run. Let me change this to use that

ludfjig · 2026-03-12T03:11:03Z

.github/workflows/Coverage.yml

+      - name: Upload LCOV coverage report
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: coverage-lcov-${{ matrix.hypervisor }}-${{ matrix.cpu }}
+          path: target/coverage/lcov.info


I think there's an option to fail the workflow if the file is not found which could be useful. Should we also using existing framework/script to create an automatic issue if this workflow fails (like we do in another workflow)?

ludfjig · 2026-03-12T03:14:17Z

.github/workflows/Coverage.yml

+      - name: Install cargo-llvm-cov
+        run: |
+          cargo install cargo-llvm-cov
+          rustup component add llvm-tools


technically this shouldn't be needed since ensure-tools will be called below I think. Do we still want it? (I am indifferent). Also, have you considred https://github.com/taiki-e/cargo-llvm-cov#on-github-actions ?

Haven't considered the gh action, good suggestion.
But I think I'll remove this and only rely on the ensure- recipe

ludfjig · 2026-03-12T03:17:07Z

Justfile

+
+# generate a text coverage summary to stdout (run `just guests` first to build guest binaries)
+coverage: ensure-cargo-llvm-cov
+    cargo llvm-cov {{ coverage-packages }}


What does llvm-cov do by default? Run all unit and integration tests?

It runs the tests (cargo test)

dblnz · 2026-03-16T12:35:24Z

This is awesome, I also tried it locally and works great. I only have one question: is this the recommended way to generate coverage reports (using cargo-llvm-cov), or are there alternative ways? I'm only asking because I am not familiar with this area.

From the research I've done, there are two other alternatives to llvm-cov: tarpaulin and grcov.
Tarpaulin uses ptrace by default (only on Linux), can be configured to use LLVM, but it has some limitations (coverage data not returned when non-zero exit, some areas of thread unsafety).
Grcov could also be a good alternative for us, it supports multiple aggregation sources, but it's slightly more complicated to set up. Given that I don't think we need that much flexibility at the moment, I haven't used this one.
I chose llvm-cov due to its accuracy (uses the llvm backend), support across OS (we could enable windows also) and its relatively simple setup.

dblnz · 2026-03-16T12:54:29Z

Its nice to see a report. Do you have thoughts on how do we ensure we are looking at this report and not just another artifact? Do we post an issue on run? post in release? Review at community meetings?

I ran it locally and I am seeing 70.42%, which quite high! Its nice to see this. I am not to worried about minor changes but would like to see it trending even or even improving over time.

About this, I think we should definitely have this in mind when we do changes. Although, I wouldn't condition any PR on it as it takes a lot to run (same time the PR validations take).
Considering we are still making fundamental changes in the codebase, this metric could change a lot.

So maybe start with having this info as part of the release artifacts is a good start? Then we could see if we need to discuss it more often.

One nice to have thing I was thinking about is to try and correlate the coverage output with the changes made in a PR so that we could have a check saying we are only testing x percent of the newly introduced code. I'll hack at this once I get some more time

dblnz · 2026-03-16T12:58:05Z

You can check out how the summary looks like at: https://github.com/hyperlight-dev/hyperlight/actions/runs/22959438794?pr=1301

dblnz · 2026-03-16T12:59:02Z

I am also looking at enabling the branch coverage report (it needs nightly).

dblnz added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Mar 11, 2026

dblnz requested review from danbugs, devigned, jprendes, ludfjig, marosset, simongdavies and syntactically as code owners March 11, 2026 14:57

Add PR trigger to test the new workflow, remove afterward

808b8eb

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

jsturtevant reviewed Mar 11, 2026

View reviewed changes

ludfjig approved these changes Mar 12, 2026

View reviewed changes

Conversation

dblnz commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsturtevant left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ludfjig Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dblnz commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dblnz commented Mar 16, 2026

Uh oh!

dblnz commented Mar 16, 2026

Uh oh!

dblnz commented Mar 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dblnz commented Mar 11, 2026 •

edited

Loading

jsturtevant left a comment •

edited

Loading

ludfjig Mar 12, 2026 •

edited

Loading

dblnz commented Mar 16, 2026 •

edited

Loading